Conventional image encoding apparatuses generate encoded data, or a bit stream, by digitizing moving picture signals input from the outside and then performing encoding processing in conformity with a predetermined image encoding scheme.
Such image encoding schemes include ISO/IEC 14496-10, Advanced Video Coding, which was recently approved as a standard (see Non-patent Document 1, ISO/IEC 14496-10 Advanced Video Coding), and one known reference model for such an AVC encoder is that of a Joint Model scheme (which will be referred to as conventional image encoding apparatus hereinbelow).
Now the configuration and operation of a conventional image encoding apparatus in which a digitized image frame is input and a bit stream is output will be described hereinbelow with reference to FIG. 1.
Referring to FIG. 1, a conventional image encoding apparatus is comprised of an original image frame buffer 1000, an MB encoding apparatus 2000, a rate control apparatus 3000, and a decoded image frame buffer 4000.
In a case that the image frame is in QCIF (Quarter Common Intermediate Format), the original image frame buffer 1000 stores therein image data as shown in FIG. 2. An image frame is divided into pixel blocks, which are referred to as MB's (Macro Blocks), each composed of 16×16 brightness pixels and 8×8 Cr/Cb color difference pixels.
The MB encoding apparatus 2000 generally encodes such MB's in a sequence of a raster scan from the upper left to the lower right of an image frame.
The rate control apparatus 3000 monitors the number of output bits in a bit stream from the MB encoding apparatus 2000 to adjust a quantization parameter to be supplied to the MB encoding apparatus 2000 for controlling the rate so that the output bit stream is brought closer to a target number of bits. In particular, when the number of bits in a bit stream becomes greater than the target number of bits, the apparatus 3000 supplies a quantization parameter indicating a larger quantization step size to the MB encoding apparatus 2000; on the contrary, when the number of bits in the bit stream becomes smaller than the target number of bits, it supplies a quantization parameter indicating a smaller quantization step size to the MB encoding apparatus 2000.
The decoded image frame buffer 4000 reads a decoded image from the MB encoding apparatus 2000 and stores it therein at the time when the MB encoding apparatus 2000 has completed encoding of one MB, for use in subsequent encoding (prediction).
Next, the internal configuration and operation of the MB encoding apparatus 2000 will be described in detail.
Referring to FIG. 1, the MB encoding apparatus 2000 is comprised of a Read apparatus 2100 (an original image MB memory 2110 and a reference image memory 2120), a prediction apparatus 2200, a Venc apparatus 2300 (a conversion/quantization apparatus 2310, an inverse-quantization/inverse-conversion apparatus 2320, and a decoded image MB memory 2330), and an EC apparatus 2400 (an entropy encoding apparatus 2410, an output buffer 2420, a control apparatus 2430, and a Context copy memory 2440).
The original image MB memory 2110 reads an image of an MB to be encoded (which will be referred to simply as original image org hereinbelow) from an image frame stored in the original image frame buffer 1000, and stores it therein.
The reference image memory 2120 reads an image required to predict and encode the MB to be encoded (which image will be referred to simply as reference image ref hereinbelow) from the decoded image frame buffer 4000, and stored it therein.
The prediction apparatus 2200 detects a prediction parameter param that allows the original image to be suitably encoded, from a reference image stored in the decoded image frame buffer 4000 and decoded image memory 2330, which will be discussed later, to produce a predicted image pred and a predictive error image pe, which will be discussed later. The prediction parameter is supplied to the entropy encoding apparatus 2410. The predictive error image pe is supplied to the conversion/quantization apparatus 2310, which will be discussed later. The predictive error image pe is added to an output of the inverse-quantization/inverse-conversion apparatus 2320, which will be discussed later, and is stored in the decoded image MB memory 2330 as decoded image.
However, in a case that original image PCM re-encoding, which will be discussed later, occurs, an original image supplied from the original image MB memory 2110 is stored in the decoded image MB memory 2330 as decoded image.
The aforementioned prediction includes two types: intra-frame prediction and inter-frame prediction. Now predicted images and predictive error images in intra-frame prediction/inter-frame prediction will be described below.
Intra-frame prediction involves referring to a past decoded image whose display time is the same as that of a current image frame to be encoded, and producing a predicted image pred using correlation between pixels within an image frame (in a spatial direction). Exemplary intra-frame prediction for a 4×4 pixel block size of a sub-divided MB is shown in FIG. 3 (see Non-patent Document 1, ISO/IEC 14496-10 Advanced Video Coding, Section 8.3: Intra prediction process, for color differences or intra frame prediction in other cases).
Intra_dir in FIG. 3 designates an in-screen prediction direction parameter indicative of a direction of intra-frame prediction. Upon selection of intra-frame prediction, the prediction apparatus 2200 produces a predicted image pred according to the in-screen prediction direction parameter intra_dir. For convenience of the following explanation, intra-frame prediction is defined as:pred=intra_prediction(ref,intra_dir)  (1),where intra_prediction( ) is a function that produces an intra-frame predicted image from a reference image ref according to an in-screen prediction direction parameter intra_dir.
On the other hand, inter-frame prediction involves referring to a past decoded image whose display time is different from that of a current image frame to be encoded, and producing a predicted image pred using correlation between image frames (in a temporal direction). Exemplary inter-frame prediction for a 16×16 pixel block is shown in FIG. 4 (see Non-patent Document 1, ISO/IEC 14496-10 Advanced Video Coding, Section 8.4: Inter prediction process, for inter-frame prediction in other pixel block sizes).
Motion vectors mv_x, mv_y in FIG. 4 are prediction parameters in inter-frame prediction. Upon selection of inter-frame prediction, the prediction apparatus 2200 produces a predicted image pred according to the motion vectors mv_x, mv_y. For convenience of the following explanation, inter-frame prediction is defined as:pred=inter_prediction(ref,mv—x,mv—y)  (2),where inter_prediction( ) is a function that produces an inter-frame predicted image from a reference image ref according to motion vectors mv_x, mv_y. For AVC, pixel precision of a motion vector is ¼ pixel.
The prediction apparatus 2200 uses a cost function (prediction evaluation value) of EQ. (3) to detect a prediction parameter param for producing the aforementioned predicted image pred. The prediction parameter as used herein refers to the in-screen prediction direction intra_dir as described above for intra-frame prediction, and the motion vectors mv_x, mv_y as described above for inter-frame prediction (see Non-patent Document 1, ISO/IEC 14496-10 Advanced Video Coding, Section 7: Syntax and semantics, for other prediction parameters).
                              Cost          ⁡                      (            param            )                          =                                            ∑                              idx                =                0                            15                        ⁢                          SATD              ⁡                              (                idx                )                                              +                                    λ              ⁡                              (                QP                )                                      ×                          R              ⁡                              (                param                )                                                                        (        3        )                                                      SATD            ⁡                          (              idx              )                                =                      0.5            ×                                          ∑                                  x                  =                  0                                3                            ⁢                                                ∑                                      y                    =                    0                                    3                                ⁢                                                                                              H                      ⁡                                              (                        idx                        )                                                              xy                                                                                                        )                            (        4        )                                          H          ⁡                      (            idx            )                          =                              [                                                            1                                                  1                                                  1                                                  1                                                                              1                                                  1                                                                      -                    1                                                                                        -                    1                                                                                                1                                                                      -                    1                                                                                        -                    1                                                                    1                                                                              1                                                                      -                    1                                                                    1                                                                      -                    1                                                                        ]                    ⁢                                                                 [                                                                  ⁢                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      0                            ,                            0                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      0                            ,                            1                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      0                            ,                            2                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      0                            ,                            3                                                    )                                                                                                                                                                                                  pe                          idx                                                ⁡                                                  (                                                      1                            ,                            0                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      1                            ,                            1                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      1                            ,                            2                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      1                            ,                            3                                                    )                                                                                                                                                                                                  pe                          idx                                                ⁡                                                  (                                                      2                            ,                            0                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      2                            ,                            1                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      2                            ,                            2                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      2                            ,                            3                                                    )                                                                                                                                                                                                  pe                          idx                                                ⁡                                                  (                                                      3                            ,                            0                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      3                            ,                            1                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      3                            ,                            2                                                    )                                                                                                                                                              pe                          idx                                                ⁡                                                  (                                                      3                            ,                            3                                                    )                                                                                                                    ]                            ⁢                                                                 [                                                                          ⁢                                                                                    1                                                                    1                                                                    1                                                                    1                                                                                                            1                                                                    1                                                                                              -                          1                                                                                                                      -                          1                                                                                                                                    1                                                                                              -                          1                                                                                                                      -                          1                                                                                            1                                                                                                            1                                                                                              -                          1                                                                                            1                                                                                              -                          1                                                                                                      ]                                                                                        (        5        )                                                      pe            idx                    ⁡                      (                          y              ,              x                        )                          =                              org            ⁡                          (                                                                    b                    ⁢                                                                                  ⁢                    4                    ⁢                                          y                      idx                                                        +                  y                                ,                                                      b                    ⁢                                                                                  ⁢                    4                    ⁢                                          x                      idx                                                        +                  x                                            )                                -                      pred            ⁡                          (                                                                    b                    ⁢                                                                                  ⁢                    4                    ⁢                                          y                      idx                                                        +                  y                                ,                                                      b                    ⁢                                                                                  ⁢                    4                    ⁢                                          x                      idx                                                        +                  x                                            )                                                          (        6        )                                                      λ            ⁡                          (              QP              )                                =                      2                                          (                                  QP                  -                  12                                )                            /              6                                      ,                            (        7        )            where QP designates a quantization parameter for an MB, idx designates an index of a 4×4 block within an MB in FIG. 2, and(b4xidx,b4yidx){0≦b4xidx≦12,0≦b4yidx≦12}designates coordinates at the upper left corner of a 4×4 block corresponding to idx within an MB.
A difference between the predicted image pred corresponding to the detected prediction parameter param and original image org is designated as a predictive error image pe (see EQ. (6)).
The description of predicted image and predictive error image in intra-frame prediction/inter-frame prediction is now concluded.
Next, the conversion/quantization apparatus 2230 frequency-converts the predicted error pe supplied from the prediction apparatus 2200 for each unitary block (which will be referred to as converted block hereinbelow) smaller than the MB to transfer it from the spatial domain into the frequency domain. The predictive error image converted into the frequency domain will be referred to as a conversion factor T.
The conversion/quantization apparatus 2230 also quantizes the conversion factor T at a quantization step size corresponding to the quantization parameter QP supplied from the rate control apparatus 3000. The quantized conversion factor is generally referred to as a converted/quantized value L. The converted/quantized value L is supplied to the inverse-quantization/inverse-conversion apparatus 2330 for subsequent encoding, and to the EC apparatus 2400 for bit stream formation.
First, a subsequent operation will be described in view of the converted/quantized value L supplied to the inverse-quantization/inverse-conversion apparatus 2330.
The inverse-quantization/inverse-conversion apparatus 2330 inversely quantizes the converted/quantized value L supplied from the conversion/quantization apparatus 2230, and inversely frequency-converts it back into the original spatial domain. The predictive error image converted back into the original spatial domain will be referred to as reconstructed predictive error image pe_rec hereinbelow.
The reconstructed predictive error image pe_rec supplied from the inverse-quantization/inverse-conversion apparatus 2330 is added with the predicted image pe supplied from the prediction apparatus 2200 (EQ. (8)), and the resulting image is stored in the decoded image MB memory 2330 as decoded image. The decoded image rec stored in the decoded image MB memory 2330 is thereafter read into the prediction apparatus 2200 and decoded image frame buffer 4000 and serves as a reference image.rec=pred+pe—rec  (8)
Next, a subsequent operation will be described in view of the converted/quantized value L supplied to the EC apparatus 2400.
The EC apparatus 2400 is comprised of an entropy encoding apparatus 2410, an output buffer 2420, a control apparatus 2430, and a Context copy memory 2440.
The entropy encoding apparatus 2410 entropy-codes input data and supplies the resulting output bits to the output buffer 2420. The control apparatus 2430 monitors the number of output bits of the entropy encoding apparatus 2410 to control the operation of the other apparatuses. The Context copy memory 2440 is a memory for storing a copy of Context data, which will be discussed later.
The operation of these apparatuses will now be described.
The control apparatus 2430 monitors the number of output bits of the entropy encoding apparatus 2410 to control the entropy encoding apparatus 2410 and output buffer 2420 through control signals (an entropy encoding control signal and an output buffer control signal).
In a case that all input data in one MB are entropy-encoded and the resulting number of output bits does not exceed an upper limit number of bits stipulated by the AVC standard (3200 bits referring to Non-patent Document 1 ISO/IEC 14496-10, Advanced Video Coding, Annex A: A.3 Levels), the control apparatus 2430 causes the bits stored in the output buffer 2420 to be output as encoded data for that MB through the output buffer control signal. As used herein, the upper limit number of bits per MB stipulated by the aforesaid standard will be referred to simply as a specified value.
On the other hand, at the moment when the number of output bits for input data of one MB entropy-encoded exceeds the specified value, the control apparatus 2430 suspends the entropy encoding apparatus 2410 through an entropy encoding control signal, and discards all bits in the output buffer 2420 through an output buffer control signal (that is, the content in the output buffer 2420 at that time is not served as encoded data). After the above-described bit discard, the control apparatus 2430 reactivates the entropy encoding apparatus 2410 through an entropy encoding control signal to re-encode input image data so that the number of bits is smaller than the specified value.
Next, the internal configuration and operation of the entropy encoding apparatus 2410 will be described.
Referring to FIG. 5, the entropy encoding apparatus 2410 is comprised of a binarization apparatus 24100, an arithmetic coder 24101, a Context Modeling apparatus 24102, and switches 24103SW, 24104SW.
The operation of the entropy encoding apparatus 2410 will be described in two cases: a case in which no re-encoding is performed, and a case in which re-encoding is performed.
First, the case in which no re-encoding is performed will be described.
The entropy encoding apparatus 2410 first saves all Context data stored in the Context Modeling apparatus 24102 into the Context copy memory 2440 immediately before entropy-encoding an MB to be processed, for handling subsequent re-encoding. Next, the entropy encoding apparatus 2410 binarizes a prediction parameter supplied from the prediction apparatus 2200 and a converted/quantized value supplied from the conversion/quantization apparatus 2310 via the switch 24103SW, by the binarization apparatus 24100 (such binarized data will be designated as a bin hereinbelow), and supplies it to the arithmetic coder 24101. At the same time, the Context Modeling apparatus 24102 supplies Context data corresponding to the aforesaid bin to the arithmetic coder 24101. The arithmetic coder 24101 arithmetically encodes the bin using the Context data, and returns the Context data update by the arithmetic encoding to the Context Modeling apparatus 24102 while writing the output bits into the output buffer 2420 via the switch 24104SW.
As used herein, Context data in AVC refers to 6-bit data of an index (State) of a probability table in which one bit of an MPS (Most Probable Symbol) and a probability value pLPS of an LPS (Least Probable Symbol) are stored for each symbol to be encoded (bin) (See Non-patent Document 1, ISO/IEC 14496-10 Advanced Video Coding, Section 9.3: CABAC parsing process for slice data, for further details). The Context data is generally initialized by a first MB within an image frame, and continually updated by subsequent arithmetic encoding.
Next, the case in which re-encoding is performed will be described.
The entropy encoding apparatus 2410 first suspends entropy encoding processing on input data. Next, Context data saved in the Context copy memory 2440 is loaded into the Context Modeling apparatus 24102. Thereafter, the binarization apparatus 24100 generates a bin of a prediction parameter indicating the start of PCM (Pulse Code Modulation), and supplies it to the arithmetic coder 24101. At the same time, the Context Modeling apparatus 24102 supplies Context data corresponding to the bin to the arithmetic coder 24101. The arithmetic coder 24101 arithmetically encodes the bin using the Context data, and returns the Context data updated by arithmetic encoding to the Context Modeling apparatus 24102 while writing the output bits into the output buffer 2420 via the switch 24104SW. After arithmetic encoding of the bin of the prediction parameter indicating the start of the PCM has been completed, the switch 24104SW is thrown, and an image stored in the original image MB memory 2110 is read for re-encoding still with PCM, and output to the output buffer 2420.
The re-encoding processing involving re-encoding with PCM an original image to which no conversion processing as described above is applied is sometimes referred to as original image PCM re-encoding (see Patent Document 1, JP-P2004-13521A, for example). Since the original image PCM re-encoding may be employed to encode an image that does not accept conversion or prediction, free from distortion using a certain number of bits smaller than a specified value, it is believed that not only the processing time for the entropy encoding apparatus but also that for the entropy decoding apparatus in the decoder side in the encoding apparatus can be ensured within a certain period of time.
The description of the operation of the EC apparatus 2400 is now concluded.
By sequentially performing MB encoding as described above, the conventional image encoding apparatus generates a bit stream.    Patent Document 1: JP-P2004-13521A    Non-patent Document 1: ISO/IEC 14496-10 Advanced Video Coding